• Capturing workflows and improving methods reproducibility (Goodman, Fanelli, and Ioannidis 2016)

<"https://www.google.com/search?q=workflow">

• Idea/question/hypotheses
• Design study
• Seek ethics board permission
• Build/borrow/buy data collection instruments
• Run study
• Analyze results
• Write-up results for presentation and/or publication

• Participants
  • Number, characteristics
• Setting(s)
  • Lab, classroom, home
• Measures or tasks
  • Self/other report
  • Observations/video or audio recording
  • Physiological measures (MRI, EEG, ECG)
  • Computer-based tasks

• Surveys
• Video/audio
• MRI, EEG, ECG
• Computer-generated data files

done.collecting.data = FALSE
while (!done.collecting.data) {
  Collect.sample()
  if (collected.sample.n >= planned.sample.n) {
    done.collecting.data = TRUE
  } else {
    done.collecting.data = FALSE
  }
}

• Clean/check data
• Merge, combine, munge data

• Intro
• Methods
• Results
  • Stats
  • Plots
• Conclusions
• References
• Data?

• Idea/question/hypotheses
• Design study
• Seek ethics board permission
• Build/borrow/buy data collection instruments
• Run study
• Analyze results
• Write-up results for presentation and/or publication

• Data collection instruments
• Running study
• Data analysis
• Study write-up

• Maximize consistency, methods reproducibility (Goodman, Fanelli, and Ioannidis 2016)
• Design of/consistent adherence to detailed experimental protocols
• Consistent, transparent workflows
• Consistent, transparent organization of data, metadata
• Minimize human/hand data entry
• Automate as much as possible

• Play & Learning Across a Year (PLAY) project wiki

• What data and metadata am I collecting?
• How does it get collected?
• Where does it go after it's collected?
• How does my non-electronic data get transferred to an electronic form?
• How do my electronic data files get cleaned, merged, munged?

• "Chain of custody" from raw data to finished results and figures
• Single command to regenerate all results and figures from raw data

http://datasci.kitzes.com/lessons/python/reproducible_workflow.html

• Create consistent structure for projects
  • Use file name conventions
• Use machine-readable file types
  • commas-separated value (.csv) vs. .xlsx
• Automate as much as possible
• Use version control

study-1/
  sub-001/
    sub-001-measure-a.txt
    sub-001-image.jpg
    sub-001-demo.csv
    sub-001-measure-b.txt
  sub-002/
    sub-002-measure-a.txt
    sub-002-image.jpg
    sub-002-demo.csv
    sub-002-measure-b.txt
  ...
  sub-00n/
    ...

study-1/
  measure-a/
    sub-001-measure-a.txt
    ...
  measure-b/
    sub-001-measure-b.txt
    ...
  image/
    sub-001-image.jpg
    sub-002-image.jpg
    ...
  demo/
    sub-001-demo.csv
    sub-002-demo.csv
    ...

study-1/
  analysis/
    data/
      sessions/
        2017-01-09-sub-001/
        ...
      aggregate/
        study-1-demo-aggregate.csv
        study-1-measure-a-aggregate.csv
        ...
    R/
    img/
    reports/
  protocol/
    code/
      my-experiment.m
    materials/
      stim-1.jpg
      stim-2.jpg
      ...

  pubs/
    presentations/
    papers/
  refs/
  grants/
    2016/
    2017/
  irb/
  mtgs/

https://nyu.databrary.org/volume/2

• Automates some of the project management involved in data analysis
  • Hat Tip (HT): Michael Hallquist
• Gilmore says: Use what you like

## Create project directory
proj.name = "tmp_proj"
if (!exists(proj.name)) {
  dir.create(path = proj.name, recursive = TRUE)
}

# Create sessions directory
sessions.dir = paste(proj.name, "analysis/sessions", sep="/")
if (!exists(sessions.dir)) {
  dir.create(path = sessions.dir, recursive = TRUE) # creates intermediate dirs
}

# Aggregate data file directory
agg.dir = paste(proj.name, "analysis/aggregate", sep="/")
if (!exists(agg.dir)) {
  dir.create(path = agg.dir, recursive = TRUE)
}

• Use consistent file/directory names
  • lowerCamelCaseIsGood.txt so is UpperCamelCase.txt
  • underscores_between_words.txt works; so do dashes-between.txt
  • avoid spaces in your file names.txt; these are not always reliably readable by all computers.
• Choose good, descriptive names

• Be consistent
• Write dates as YYYY-MM-DD.
• Fill-in all cells
• One thing in a cell
• Make your data a rectangle
• Create a data dictionary

• No calculations in raw data files
• No font or color to highlight data
• Make back-ups
• Validate data to avoid data entry errors
• Save data in plain text files (comma or tab-delimited)

• Data scientists (that's you!) spend a lot of time just cleaning data
• http://www.infoworld.com/article/3047584/big-data/hottest-job-data-scientists-say-theyre-still-mostly-digital-janitors.html

• study-1-demo-agg.csv contains
  • subID, sex, ageYrs, favColor
• study-1-rt-agg.csv contains
  • subID, condition, rt

subID,sex,ageYrs,favColor
001,m,53,green
002,f,51,blue
003,f,23,red
004,m,25,aqua

Don't put spaces between variables in comma-separated value (.csv) files. Also, make sure to add a final line feed/enter character.

subID,condition,rt
001,upright,250
001,inverted,300
002,upright,225
002,inverted,290
003,upright,270
003,inverted,230
004,upright,210
004,inverted,240

# read data files, first row (header) contains variable names
demo <- read.csv(file = "csv/study-1-demo-agg.csv", header = TRUE)
rt <- read.csv(file = "csv/study-1-rt-agg.csv", header = TRUE)

# merge and print
merged <- merge(demo, rt, by = "subID")
merged

##   subID sex ageYrs favColor condition  rt
## 1     1   m     53    green   upright 250
## 2     1   m     53    green  inverted 300
## 3     2   f     51     blue   upright 225
## 4     2   f     51     blue  inverted 290
## 5     3   f     23      red   upright 270
## 6     3   f     23      red  inverted 230
## 7     4   m     25     aqua   upright 210
## 8     4   m     25     aqua  inverted 240

• Variables in columns
• Observations in rows
• Ok/better to repeat values in columns
  • subID,trial,rt
  • 001,1,300
  • 001,2,345
  • 002,2,327
  • 003,3,429

• Think like a computer!
• Plan your work; work your plan
• Consistency, standard formats
• Tidy data
• Haven't said anything about "openness"…yet

• Data Carpentry, http://www.datacarpentry.org/
• Software Carpentry, https://software-carpentry.org/
• Open Science Framework (OSF), http://osf.io
• R for Data Science, http://r4ds.had.co.nz/

• Pick one of the recommended elements from Table 1 in Munafò, et al. (2017). Evaluate the recommendation. Do you agree that it would mitigate one or more threats to reproducibility. Why or why not? Do you agree with the assessment about the degree to which stakeholders have adopted the recommended practice?
• Edit/create a text-based workflow for an active project you are working on.
  • Annote the workflow to indicate where it could be made more reproducible, transparent.